{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "965efe13-1699-4542-9703-aa7ead0c0ba2",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Using deepchecks with an H2O Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53a383a0-63f6-4545-80fe-4ac796804347",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "[H2O](https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html) is a wildly used AutoML framework, written in Java but with multiple fully implemented API libraries, including one in python. In this example, we'll be demonstrating how to effortlessly test your H2O model with deepchecks in the exact same way that you can test any sklearn model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a55b5012-0f1f-4500-bfdf-7514bccbc23b",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Building an H2O Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ea2c4db-e689-4ad6-9f6a-a622880f6fd9",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "First we'll load our dataset, using the adult dataset that can be downloaded from the [UCI machine learning repository](http://archive.ics.uci.edu/ml>) and is also available in `deepchecks.tabular.datasets`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d091d3d4-6cd6-4d38-86cc-097e44ec6fee",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from deepchecks.tabular.datasets.classification import adult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fbed1df-2528-4797-843f-740e7a8c3178",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "train_df, test_df = adult.load_data(data_format='Dataframe', as_train_test=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4d8d8b",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "install h2o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab644803",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install h2o"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cb2f75-8765-407f-8ab7-ac323f9e0e68",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we'll build an H2O [H2OGradientBoostingEstimator](https://docs.h2o.ai/h2o/latest-stable/h2o-py/docs/modeling.html#h2o.estimators.gbm.H2OGradientBoostingEstimator) based on this dataset split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9e0c151-b09c-43bb-b7f2-331a0408349d",
   "metadata": {
    "scrolled": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import h2o\n",
    "h2o.no_progress()\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a956599-1b18-4046-a2c6-ccacd869e030",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "H2O models can only train and predict on H2OFrame objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fa22d7d-67de-48c1-af15-304391b3deb1",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "train_h2o = h2o.H2OFrame(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85198fe9-60dd-494c-a842-20bb74aab463",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gbm = h2o.estimators.gbm.H2OGradientBoostingEstimator()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052c0004-2b48-43ff-ac9d-a981f82d3a02",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We must tell the H2O model what are the feature columns and what is the name of the label column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9879547f-7088-47d3-8aef-8c8f352ada51",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "feature_cols = adult._FEATURES\n",
    "print(feature_cols)\n",
    "label_col = adult._target\n",
    "print(label_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcdc51cf-cf65-4305-992e-365032e7df8e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gbm.train(x=feature_cols, y=label_col, training_frame=train_h2o, verbose=False)\n",
    "print('Trained!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "528122ca-4ac0-4026-b088-681db1c00218",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "To compare to our new H2O model, we'll download the default scikit-learn model (a pipeline containing a naive Random Forest Classifier) for this task from deepchecks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f866539a-2729-47ba-ab75-4637b79604a8",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "skmodel = adult.load_fitted_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b4e23c4-5af3-4a9a-adae-656a0c02afb4",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Running deepchecks on H2O"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ccf033b-bedb-4d58-bc35-a4a422f101ca",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Before we can run deepchecks on our H2O model, we have one step remaining - bridging between the API of the H2O model and the scikit-learn API that deepchecks expects.\n",
    "\n",
    "We have created a wrapper to our H2O model, that implements the required API for deepchecks according the [supported models](https://docs.deepchecks.com/stable/user-guide/tabular/supported_models.html) guide. This wrapper is available from `deepchecks.tabular.integrations.h2o`.\n",
    "Generally the wrapper model will contain 2 functions in case of a classification problem: the ``predict`` and the ``predict_proba`` functions that will be called by deepchecks. In addition, it is also recommended to define a way to get the feature importances of the model, implemented here using the h2o model's internal calculation. Read more about feature importance handling in the [feature importance](https://docs.deepchecks.com/dev/user-guide/tabular/feature_importance.html) guide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0066f11-a806-4a9a-ad66-75f4e1633a29",
   "metadata": {
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from deepchecks.tabular.integrations.h2o import H2OWrapper\n",
    "h2o_dc_gbm = H2OWrapper(gbm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90a5659c-a659-4e1c-9057-e9568ff13c09",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we can run deepchecks! We'll start by defining the [deepchecks Dataset](https://docs.deepchecks.com/stable/user-guide/tabular/dataset_object.html) objects and comparing the two models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f330cbfe-1ebe-4256-ba12-cd00e46f214f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from deepchecks.tabular import Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89ba8ae1-af92-4b9c-b8d4-e0420aee1479",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "train_ds = Dataset(train_df, label=label_col, features=feature_cols, cat_features=adult._CAT_FEATURES)\n",
    "test_ds = Dataset(test_df, label=label_col, features=feature_cols, cat_features=adult._CAT_FEATURES)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c291481f-d103-44a3-aa70-d84181707e06",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Comparing model performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19bf9971-0a4d-4df1-b6ac-d1b1cb001e11",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from deepchecks.tabular.checks import MultiModelPerformanceReport"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e5a08a0-2966-438a-8073-de06d7ff6791",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "MultiModelPerformanceReport().run(train_ds, test_ds, [h2o_dc_gbm, skmodel])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39af60da-7adc-49b7-8f6a-105c2e16af02",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We can see that from an overall performance standpoint (considering the f1 metric), the H2O model achieves superior performance compared to the sklearn model, especially in the smaller class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30c92c88-a623-4f58-9779-9bf4f52b57fa",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Further model performance evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e90cdfd-63f5-4ddd-acb8-4fd0aa0ff3a7",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We next wish to get a more in depth evaluation of the H2O model. For example, we can use the SegmentPerformance check to compare the performance of the two models across segments of the sex and relationship status features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67826553-2667-4fc2-a2ea-b8cc0f55dad6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from deepchecks.tabular.checks import SegmentPerformance\n",
    "from sklearn.utils import parallel_backend\n",
    "with parallel_backend('threading', n_jobs=2):\n",
    "    result_h2o = SegmentPerformance(feature_1='sex', feature_2='relationship').run(test_ds, h2o_dc_gbm)\n",
    "    \n",
    "result_skl = SegmentPerformance(feature_1='sex', feature_2='relationship').run(test_ds, skmodel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9f5f992-dbdb-4da2-a5df-30e17745132f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "result_h2o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bb4bffd-fc32-4717-939b-96328eaa0df9",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "result_skl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64c1d871-f33a-4a66-b4a5-06afa4e9fd5c",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We see that the scikit-learn model, apart from being worse overall, is specifically worse (a drop in accuracy of 0.6) at predicting the income of females with a relationship status of \"Others\"."
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Wrapping up"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We've demonstrated how you can take your h2o model as is and use deepchecks to evaluate the model and compare it to other models, but of course the sky is the limit! Once you've wrapped your h2o model with the H2OWrapper, you can use it with any other deepchecks check and suites, the same way you can use your sklearn models.\n",
    "Those of you how want to dive a bit deeper can have a look at the wrapper object itself, located at `deepchecks.tabular.integrations.h2o`, and see that implementing such a wrapper is barely any work at all! The same can be done for nearly any other custom model out there, making deepchecks easy to use with any tabular use-case you may encounter."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}